Skip to content

NUTCH-3139 protocol-okhttp: add support for zstd content-encoding - #877

Merged
sebastian-nagel merged 1 commit into
apache:masterfrom
sebastian-nagel:NUTCH-3139
Dec 18, 2025
Merged

sebastian-nagel merged 1 commit into
apache:masterfrom
sebastian-nagel:NUTCH-3139

Conversation

@sebastian-nagel

Copy link
Copy Markdown
Contributor
  • upgrade to OkHttp 5.3.2
  • enable support for zstd content-encoding

Meta sites already support zstd content-encoding:

$> $NUTCH_HOME/bin/nutch parsechecker https://www.llama.com/faq/
...
  _request_ = GET /faq/ HTTP/2
...
Accept-Encoding: zstd, br, gzip
...
  _response.headers_ = HTTP/2 200 
vary: Accept-Encoding
content-encoding: zstd
...

(with http.useHttp2, store.http.request and store.http.headers set to true)

- upgrade to OkHttp 5.3.2
- enable support for zstd content-encoding

@lewismc lewismc left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice. Also good to get a major dependency upgrade for the okhttp dependency.

@sebastian-nagel
sebastian-nagel merged commit 00bf8c4 into apache:master Dec 18, 2025
6 checks passed
@sebastian-nagel
sebastian-nagel deleted the NUTCH-3139 branch December 18, 2025 09:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants